{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallel Processing with Metakernel\n",
    "\n",
    "Metakernel uses the `ipyparallel` system for running code in parallel. This notebook demonstrates the process using the kernel Calysto Scheme. However, other Metakernel-based kernels may also work. The kernel needs to be able to return values, and implement kernel.set_variable() and kernel.env.\n",
    "\n",
    "Before opening a notebook, you should run the following command. This example starts 10 nodes in the cluster:\n",
    "\n",
    "```shell\n",
    "ipcluster start -n 10 --ip=10.0.0.190\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then initiate the parallel communication using the `%parallel` line magic. In this example, the 10 distributed kernels will be the same type of kernel as the host kernel, namely Calysto Scheme. We call %parallel with the name of the module and the name of the class of the kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "got unknown result: 834e76c8-bee8d3c516994eed98c32d1c\n"
     ]
    }
   ],
   "source": [
    "%parallel calysto_scheme CalystoScheme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see if everything is working, we can \"parallel execute\" using the %%px cell magic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#10(0 1 4 9 16 25 36 49 64 81)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%px\n",
    "\n",
    "(* cluster_rank cluster_rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't want to return the results, but rather save them in a variable, use the `--set_variable` flag followed by the name of the variable. That sets the variable in the host kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%px --set_variable results\n",
    "\n",
    "(* cluster_rank cluster_rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we then have access to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#10(0 1 4 9 16 25 36 49 64 81)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use the Metakernel `%%%` triple magic to initiate \"sticky magics\". This means that the `%%px` magic will be active for the following cells, until we turn it off. So, all of the next few cells will be sent to all of the cluster kernels. The `-e` means to also evaluate the cell in the host kernel as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "px added to session magics.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "#10(10 10 10 10 10 10 10 10 10 10)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%%px -e\n",
    "\n",
    "cluster_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define some useful Scheme code for creating a `for` syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "(define-syntax for\n",
    "  [(for ?exp times do . ?bodies)\n",
    "   (for-repeat ?exp (lambda () . ?bodies))]\n",
    "  [(for ?var in ?exp do . ?bodies)\n",
    "   (for-iterate1 ?exp (lambda (?var) . ?bodies))]\n",
    "  [(for ?var at (?i) in ?exp do . ?bodies)\n",
    "   (for-iterate2 0 ?exp (lambda (?var ?i) . ?bodies))]\n",
    "  [(for ?var at (?i ?j . ?rest) in ?exp do . ?bodies)\n",
    "   (for ?var at (?i) in ?exp do\n",
    "    (for ?var at (?j . ?rest) in ?var do . ?bodies))])\n",
    "\n",
    "(define for-repeat\n",
    "  (lambda (n f)\n",
    "    (if (< n 1)\n",
    "    'done\n",
    "    (begin\n",
    "      (f)\n",
    "      (for-repeat (- n 1) f)))))\n",
    "\n",
    "(define for-iterate1\n",
    "  (lambda (values f)\n",
    "    (if (null? values)\n",
    "    'done\n",
    "    (begin\n",
    "      (f (car values))\n",
    "      (for-iterate1 (cdr values) f)))))\n",
    "\n",
    "(define for-iterate2\n",
    "  (lambda (i values f)\n",
    "    (if (null? values)\n",
    "    'done\n",
    "    (begin\n",
    "      (f (car values) i)\n",
    "      (for-iterate2 (+ i 1) (cdr values) f)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's solve a problem in parallel! For this example, we will compute the Mandelbrot set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "(define MAX 255)\n",
    "(define n  30)\n",
    "(define xc -0.5)\n",
    "(define yc 0)\n",
    "(define size 2)\n",
    "\n",
    "(define mandelbrot\n",
    "  (lambda (z0 limit)\n",
    "    (let loop ((z z0)\n",
    "               (t 0))\n",
    "      (if (> (abs z) 2.0)\n",
    "          t\n",
    "          (if (> t limit)\n",
    "              limit\n",
    "              (loop (+ (* z z) z0) (+ t 1)))))))\n",
    "\n",
    "(define alphabet \"@MHWRmEBSQKUGgqyp$8XDPFwdbkA&0ZTNhe9654YV*Cnsyza%3OLxo2JufIrc][vt}{71lji?|+)(=;-_!~:/,^.` \")\n",
    "\n",
    "(define ascii\n",
    "  (lambda (i)\n",
    "      (get-item alphabet (int (* (/ i 256) 90)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a data structure to store our results, a 2D matrix (vector of vectors):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "(define matrix \n",
    "  (list->vector \n",
    "    (map list->vector \n",
    "         (map range \n",
    "              (map (lambda (v) n) \n",
    "                   (range n))))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And an easy way to get/set the 2D location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "(define mget\n",
    "  (lambda (matrix x y)\n",
    "    (vector-ref (vector-ref matrix x) y)))\n",
    "\n",
    "(define mset!\n",
    "  (lambda (matrix x y value)\n",
    "    (vector-set! (vector-ref matrix x) y value)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get location (10, 10) from all of the matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#10(10 10 10 10 10 10 10 10 10 10)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(mget matrix 10 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good, they are all \"10\". Now let's set all of these locations to the string \"a\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "(mset! matrix 10 10 \"a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#10(\"a\" \"a\" \"a\" \"a\" \"a\" \"a\" \"a\" \"a\" \"a\" \"a\")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(mget matrix 10 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, good. everything is working. Let's now run some code just on the host. So, we remove the sticky magics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%%px removed from session magics.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%%px"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How long does it take to compute a small picture of the Mandelbrot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " ``````````````````.^^^```````\n",
      "```````````````````.^~..``````\n",
      "``````````````````.;/@^^``````\n",
      "````````````````...l@@@^.`````\n",
      "``````````````....^!@@@^....``\n",
      "`````````````.~^^-o/@@:,^^../`\n",
      "````````````..:@/@@@@@@@@/^/:.\n",
      "```````````..^/@@@@@@@@@@@2@,.\n",
      "````.````...~:@@@@@@@@@@@@@~^.\n",
      "```.^.......^;@@@@@@@@@@@@@@,^\n",
      "``..,^^/^^.^!@@@@@@@@@@@@@@@@~\n",
      "``..^-:!!!^^)@@@@@@@@@@@@@@@@^\n",
      "``..,;@@@@1/@@@@@@@@@@@@@@@@@=\n",
      "..^^~@@@@@@!@@@@@@@@@@@@@@@@@.\n",
      ".^!;@@@@@@@@@@@@@@@@@@@@@@@@^.\n",
      "@@@@@@@@@@@@@@@@@@@@@@@@@@@,..\n",
      ".^!;@@@@@@@@@@@@@@@@@@@@@@@@^.\n",
      "..^^~@@@@@@!@@@@@@@@@@@@@@@@@.\n",
      "``..,;@@@@1/@@@@@@@@@@@@@@@@@=\n",
      "``..^-:!!!^^)@@@@@@@@@@@@@@@@^\n",
      "``..,^^/^^.^!@@@@@@@@@@@@@@@@~\n",
      "```.^.......^;@@@@@@@@@@@@@@,^\n",
      "````.````...~:@@@@@@@@@@@@@~^.\n",
      "```````````..^/@@@@@@@@@@@2@,.\n",
      "````````````..:@/@@@@@@@@/^/:.\n",
      "`````````````.~^^-o/@@:,^^../`\n",
      "``````````````....^!@@@^....``\n",
      "````````````````...l@@@^.`````\n",
      "``````````````````.;/@^^``````\n",
      "```````````````````.^~..``````\n",
      "Time: 75.23063731193542 seconds.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "done"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "(for row in (range n) do\n",
    "    (for col in (range n) do\n",
    "         (let* ((x0 (+ (- xc (/ size 2)) (* size (/ col n))))\n",
    "                (y0 (+ (- yc (/ size 2)) (* size (/ row n))))\n",
    "                (z0 (complex x0 y0))\n",
    "                (gray (- MAX (mandelbrot z0 MAX))))\n",
    "            (printf \"~a\" (ascii gray))))\n",
    "    (printf \"~%\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, that took 75 seconds on my computer. Note that this code could easily be parallelized by having each node in the cluster do a portion of the problem. \n",
    "\n",
    "Now, we execute the following cell on all nodes, but note that each node will do a different portion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: 19.94576096534729 seconds.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "%%px --set_variable results\n",
    "\n",
    "(define portion (int (/ n cluster_size)))\n",
    "\n",
    "(for col in (range n) do\n",
    "    (for row in (range (* cluster_rank portion) \n",
    "                       (* (+ cluster_rank 1) portion)) do\n",
    "         (let* ((x0 (+ (- xc (/ size 2)) (* size (/ col n))))\n",
    "                (y0 (+ (- yc (/ size 2)) (* size (/ row n))))\n",
    "                (z0 (complex x0 y0))\n",
    "                (gray (- MAX (mandelbrot z0 MAX))))\n",
    "            (mset! matrix col row (ascii gray))\n",
    "            (printf \".\")))\n",
    "     (printf \"\\n\"))\n",
    "\n",
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That went much faster, but not 10 times faster. Mostly because I really don't have 10 CPUs on my computer, so the nodes were being shared across fewer CPUs. There was also a little overhead time in collecting the results. But still, 20 seconds is much faster than 75 seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did the cluster produce the same results as a single computer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " ``````````````````.^^^```````\n",
      "```````````````````.^~..``````\n",
      "``````````````````.;/@^^``````\n",
      "````````````````...l@@@^.`````\n",
      "``````````````....^!@@@^....``\n",
      "`````````````.~^^-o/@@:,^^../`\n",
      "````````````..:@/@@@@@@@@/^/:.\n",
      "```````````..^/@@@@@@@@@@@2@,.\n",
      "````.````...~:@@@@@@@@@@@@@~^.\n",
      "```.^.......^;@@@@@@@@@@@@@@,^\n",
      "``..,^^/^^.^!@@@@@@@@@@@@@@@@~\n",
      "``..^-:!!!^^)@@@@@@@@@@@@@@@@^\n",
      "``..,;@@@@1/@@@@@@@@@@@@@@@@@=\n",
      "..^^~@@@@@@!@@@@@@@@@@@@@@@@@.\n",
      ".^!;@@@@@@@@@@@@@@@@@@@@@@@@^.\n",
      "@@@@@@@@@@@@@@@@@@@@@@@@@@@,..\n",
      ".^!;@@@@@@@@@@@@@@@@@@@@@@@@^.\n",
      "..^^~@@@@@@!@@@@@@@@@@@@@@@@@.\n",
      "``..,;@@@@1/@@@@@@@@@@@@@@@@@=\n",
      "``..^-:!!!^^)@@@@@@@@@@@@@@@@^\n",
      "``..,^^/^^.^!@@@@@@@@@@@@@@@@~\n",
      "```.^.......^;@@@@@@@@@@@@@@,^\n",
      "````.````...~:@@@@@@@@@@@@@~^.\n",
      "```````````..^/@@@@@@@@@@@2@,.\n",
      "````````````..:@/@@@@@@@@/^/:.\n",
      "`````````````.~^^-o/@@:,^^../`\n",
      "``````````````....^!@@@^....``\n",
      "````````````````...l@@@^.`````\n",
      "``````````````````.;/@^^``````\n",
      "```````````````````.^~..``````\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "done"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(define portion (int (/ n cluster_size)))\n",
    "\n",
    "(for row in (range n) do\n",
    "  (for col in (range n) do\n",
    "     (let ((vec (// row portion)))\n",
    "         (printf \"~a\" (mget (vector-ref results vec) col row))))\n",
    "    (printf \"\\n\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Calysto Scheme 3",
   "language": "scheme",
   "name": "calysto_scheme"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "scheme"
   },
   "mimetype": "text/x-scheme",
   "name": "scheme",
   "pygments_lexer": "scheme"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
